Context and Memory in Multimedia Content Analysis

ors: High-level structure and summarization Researchers have considered a variety of video summarization methods—that is, methods for condensing content into a shorter descriptive form of the original content.3 Video skim, for example, is a temporally condensed form of the video stream that should preserve the most important information. Another form of summary, video highlights, aims to include the video’s most important events. Multimedia video summary is a collection of audio, visual, and text segments that preserve the underlying video’s essence and structure (for example, pictorial summary, storyboards, and surface summary). Video programs such as news, talk shows, game shows, and sports programs have an internal structure and detectable format. These programs are conducive to higher-level structure detection and analysis, which enables program content transparency and gives users direct overview of and access to the program’s meaningful modules. To this end, colleagues and I developed a summarization system for processing incoming video, extracting and analyzing closed-caption text, determining program segment and commercial break boundaries, and extracting program summaries from complete broadcasts to enable video transparency.3 The system consists of a transcript extractor, program type classifier, cue extractor, knowledge database, temporal database, and inference engine. The final automatically derived summary contains the host and guest names, the guests’ field of activity, and the topics of conversation.